Optical See-through Head-mounted Display Research Articles

Non-deterministic method for semi-automatic calibration of smartphone-based OST HMDs

An Augmented Reality (AR) system must show real and virtual elements as if they coexisted in the same environment. The tridimensional aligment (registration) is particularly challenging on specific hardware configurations such as Head Mounted Displays (HMDs) that use Optical See-Through (OST) technology. In general, the calibration of HMDs uses deterministic optimization methods. However, non-deterministic methods have been proposed in the literature with promising results in distinct research areas. In this work, we developed a non-deterministic optimization method for the semi-automatic calibration of smartphone-based OST HMDs. We tested simulated annealing, evolutionary strategy, and particle swarm algorithms. We also developed a system for calibration and evaluated it through an application that aligned a virtual object in an AR environment. We evaluated our method using the Mean Squared Error (MSE) at each calibration step, considering the difference between the ideal/observed positions of a set of reference points and those estimated from the values determined for the calibration parameters. Our results show an accurate OST HMD calibration for the peripersonal space, with similar MSEs for the three tested algorithms.

Real-time ultrasound-guided thoracentesis simulation using an optical see-through head-mounted display: a proof-of-concept study.

This study aimed to examine the feasibility and potential benefits of an optical see-through head-mounted display (OST-HMD) during real-time ultrasound-guided thoracentesis simulations. Six physicians performed a thoracentesis simulation using an OST-HMD and a wireless image transmission system. The time required, puncture needle visibility, pleural fluid collection success rate, and head movement during the procedure using a smart glass equipped with an inertial measurement unit were all recorded and compared with and without the HMD. Study participants successfully extracted effusions in all procedures. The use of OST-HMD did not significantly affect the time of the procedure, but notably decreased the horizontal and vertical head movements during the procedure. The study demonstrated the feasibility of using an OST-HMD in a simulated real-time ultrasound-guided thoracentesis procedure and showed the potential of HMD in thoracentesis to improve ergonomics and accuracy. Further research is necessary to confirm these findings.

Development of a Compact Optical See-Through Head-Mounted Display With Dual Focal Planes and Spatial Multiplexing Using Freeform Prisms

Development of a Compact Optical See-Through Head-Mounted Display With Dual Focal Planes and Spatial Multiplexing Using Freeform Prisms

Trajectory-based alignment for optical see-through HMD calibration

AbstractIn order to align the virtual and real content precisely through augmented reality devices, especially in optical see-through head-mounted displays (OST-HMD), it is necessary to calibrate the device before using it. However, most existing methods estimated the parameters via 3D-2D correspondences based on the 2D alignment, which is cumbersome, time-consuming, theoretically complex, and results in insufficient robustness. To alleviate this issue, in this paper, we propose an efficient and simple calibration method based on the principle of directly calculating the projection transformation between virtual space and the real world via 3D-3D alignment. The proposed method merely needs to record the motion trajectory of the cube-marker in the real and virtual world, and then calculate the transformation matrix between the virtual space and the real world by aligning the two trajectories in the observed view. There are two advantages associated with the proposed method. First, the operation is simple. Theoretically, the user only needs to perform four alignment operations for calibration without changing the rotation variation. Second, the trajectory can be easily distributed throughout the entire observation view, resulting in more robust calibration results. To validate the effectiveness of the proposed method, we conducted extensive experiments on our self-built optical see-through head-mounted display (OST-HMD) device. The experimental results show that the proposed method can achieve better calibration results than other calibration methods.

Text Entry Performance and Situation Awareness of a Joint Optical See-Through Head-Mounted Display and Smartphone System.

Optical see-through head-mounted displays (OST HMDs) are a popular output medium for mobile Augmented Reality (AR) applications. To date, they lack efficient text entry techniques. Smartphones are a major text entry medium in mobile contexts but attentional demands can contribute to accidents while typing on the go. Mobile multi-display ecologies, such as combined OST HMD-smartphone systems, promise performance and situation awareness benefits over single-device use. We study the joint performance of text entry on mobile phones with text output on optical see-through head-mounted displays. A series of five experiments with a total of 86 participants indicate that, as of today, the challenges in such a joint interactive system outweigh the potential benefits.

StainedSweeper: Compact, Variable-Intensity Light-Attenuation Display with Sweeping Tunable Retarders.

Light Attenuation Displays (LADs) are a type of Optical See-Through Head-Mounted Display (OST-HMD) that present images by attenuating incoming light with a pixel-wise polarizing color filter. Although LADs can display images in bright environments, there is a trade-off between the number of Spatial Light Modulators (SLMs) and the color gamut and contrast that can be expressed, making it difficult to achieve both high-fidelity image display and a small form factor. To address this problem, we propose StainedSweeper, a LAD that achieves both the wide color gamut and the variable intensity with a single SLM. Our system synchronously controls a pixel-wise Digital Micromirror Device (DMD) and a nonpixel polarizing color filter to pass light when each pixel is the desired color. By sweeping this control at high speed, the human eye perceives images in a time-multiplexed, integrated manner. To achieve this, we develop the OST-HMD design using a reflective Solc filter as a polarized color filter and a color reproduction algorithm based on the optimization of the time-multiplexing matrix for the selected primary color filters. Our proof-of-concept prototype showed that our single SLM design can produce subtractive images with variable contrast and a wider color gamut than conventional LADs.

GlassMessaging

Communicating with others while engaging in simple daily activities is both common and natural for people. However, due to the hands- and eyes-busy nature of existing digital messaging applications, it is challenging to message someone while performing simple daily activities. We present GlassMessaging, a messaging application on Optical See-Through Head-Mounted Displays (OHMDs), to support messaging with voice and manual inputs in hands- and eyes-busy scenarios. GlassMessaging is iteratively developed through a formative study identifying current messaging behaviors and challenges in common multitasking with messaging scenarios. We then evaluated this application against the mobile phone platform on varying texting complexities in eating and walking scenarios. Our results showed that, compared to phone-based messaging, GlassMessaging increased messaging opportunities during multitasking due to its hands-free, wearable nature, and multimodal input capabilities. The affordance of GlassMessaging also allows users easier access to voice input than the phone, which thus reduces the response time by 33.1% and increases the texting speed by 40.3%, with a cost in texting accuracy of 2.5%, particularly when the texting complexity increases. Lastly, we discuss trade-offs and insights to lay a foundation for future OHMD-based messaging applications.

Usability evaluation of augmented reality visualizations on an optical see-through head-mounted display for assisting machine operations

This study explores the effect of using different visual information overlays and guiding arrows on a machine operation task with an optical see-through head-mounted display (OST-HMD). Thirty-four participants were recruited in the experiment. The independent variables included visual information mode (text, animation, and mixed text and animation) and the use of guiding arrows (with and without arrows). In addition, gender difference was also an objective of this study. The task performance indicators were determined based on task completion time and error counts as well as subjective measures (system usability scale, NASA task load index, and immersion scale). This study used the mixed analysis of variance design to evaluate the main and interaction effects. The results showed that males performed better when using the mixed text and animation mode. Females performed better when using the text mode. In addition, using the mixed text and animation mode demonstrated the best outcome in system usability scale and NASA task load index. For the use of guiding arrows, the task completion time was reduced and the system usability scale, NASA task load index, and immersion scale showed positive effects.

Effects of Luminance Contrast on Depth Perception in Optical See-Through Augmented Reality

ABSTRACT Augmented reality (AR) technology presents the rendered virtual object in the real world whose depth cues can assist the depth estimation of the virtual object. For many AR applications, the successful interaction between the real world and the virtual object relies on the accuracy of depth perception. In the past, the research of depth perception in AR mainly focused on the impact of virtual object properties, including the size, color, and shading model. In this study, an experiment has been conducted to investigate the influence of ambient luminance, virtual object luminance, and shading model which is a luminance-related parameter on depth perception using an optical see-through head-mounted display. Results show that a higher luminance contrast between the real scene and the virtual object leads to a lower accuracy of depth perception in AR.

Effects of Optical See-Through Head-Mounted Display Use for Simulated Laparoscopic Surgery

This study investigated the influence of using optical see-through head-mounted displays (OST-HMDs) as a replacement for conventional monitors for an extended duration, as required for laparoscopic surgery. We developed the experiment scenario including two displays (surgical field and patient vitals) in three different combinations of the HMD and conventional monitors. An experiment was conducted to compare the task performance, situation awareness, physical discomfort, and perceived workload in the three different display settings. From this study, it was shown that even though the OST-HMD resulted in higher physical discomfort and lower self-perceived performance rating, the surgical task performance, and situation awareness were not worse, and in some conditions even better than conventional monitors. The main obstacle to using OST-HMDs to replace conventional monitors in laparoscopic surgery would be the weight of the HMDs and the head discomfort caused by wearing them for a long time.

An autonomous X-ray image acquisition and interpretation system for assisting percutaneous pelvic fracture fixation.

Percutaneous fracture fixation involves multiple X-ray acquisitions to determine adequate tool trajectories in bony anatomy. In order to reduce time spent adjusting the X-ray imager's gantry, avoid excess acquisitions, and anticipate inadequate trajectories before penetrating bone, we propose an autonomous system for intra-operative feedback that combines robotic X-ray imaging and machine learning for automated image acquisition and interpretation, respectively. Our approach reconstructs an appropriate trajectory in a two-image sequence, where the optimal second viewpoint is determined based on analysis of the first image. A deep neural network is responsible for detecting the tool and corridor, here a K-wire and the superior pubic ramus, respectively, in these radiographs. The reconstructed corridor and K-wire pose are compared to determine likelihood of cortical breach, and both are visualized for the clinician in a mixed reality environment that is spatially registered to the patient and delivered by an optical see-through head-mounted display. We assess the upper bounds on system performance through in silico evaluation across 11 CTs with fractures present, in which the corridor and K-wire are adequately reconstructed. In post hoc analysis of radiographs across 3 cadaveric specimens, our system determines the appropriate trajectory to within 2.8 ± 1.3mm and 2.7 ± 1.8[Formula: see text]. An expert user study with an anthropomorphic phantom demonstrates how our autonomous, integrated system requires fewer images and lower movement to guide and confirm adequate placement compared to current clinical practice. Code and data are available.

Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capable.

The occlusion-capable optical see-through head-mounted display (OC-OSTHMD) is actively developed in recent years since it allows mutual occlusion between virtual objects and the physical world to be correctly presented in augmented reality (AR). However, implementing occlusion with the special type of OSTHMDs prevents the appealing feature from the wide application. In this paper, a novel approach for realizing mutual occlusion for common OSTHMDs is proposed. A wearable device with per-pixel occlusion capability is designed. OSTHMD devices are upgraded to be occlusion-capable by attaching the device before optical combiners. A prototype with HoloLens 1 is built. The virtual display with mutual occlusion is demonstrated in real-time. A color correction algorithm is proposed to mitigate the color aberration caused by the occlusion device. Potential applications, including the texture replacement of real objects and the more realistic semi-transparent objects display, are demonstrated. The proposed system is expected to realize a universal implementation of mutual occlusion in AR.

Using augmented reality to cue obstacles for people with low vision.

Detecting and avoiding obstacles while navigating can pose a challenge for people with low vision, but augmented reality (AR) has the potential to assist by enhancing obstacle visibility. Perceptual and user experience research is needed to understand how to craft effective AR visuals for this purpose. We developed a prototype AR application capable of displaying multiple kinds of visual cues for obstacles on an optical see-through head-mounted display. We assessed the usability of these cues via a study in which participants with low vision navigated an obstacle course. The results suggest that 3D world-locked AR cues were superior to directional heads-up cues for most participants during this activity.

Virtual Big Heads in Extended Reality: Estimation of Ideal Head Scales and Perceptual Thresholds for Comfort and Facial Cues

Extended reality (XR) technologies, such as virtual reality (VR) and augmented reality (AR), provide users, their avatars, and embodied agents a shared platform to collaborate in a spatial context. Although traditional face-to-face communication is limited by users’ proximity, meaning that another human’s non-verbal embodied cues become more difficult to perceive the farther one is away from that person, researchers and practitioners have started to look into ways to accentuate or amplify such embodied cues and signals to counteract the effects of distance with XR technologies. In this article, we describe and evaluate the Big Head technique, in which a human’s head in VR/AR is scaled up relative to their distance from the observer as a mechanism for enhancing the visibility of non-verbal facial cues, such as facial expressions or eye gaze. To better understand and explore this technique, we present two complimentary human-subject experiments in this article. In our first experiment, we conducted a VR study with a head-mounted display to understand the impact of increased or decreased head scales on participants’ ability to perceive facial expressions as well as their sense of comfort and feeling of “uncannniness” over distances of up to 10 m. We explored two different scaling methods and compared perceptual thresholds and user preferences. Our second experiment was performed in an outdoor AR environment with an optical see-through head-mounted display. Participants were asked to estimate facial expressions and eye gaze, and identify a virtual human over large distances of 30, 60, and 90 m. In both experiments, our results show significant differences in minimum, maximum, and ideal head scales for different distances and tasks related to perceiving faces, facial expressions, and eye gaze, and we also found that participants were more comfortable with slightly bigger heads at larger distances. We discuss our findings with respect to the technologies used, and we discuss implications and guidelines for practical applications that aim to leverage XR-enhanced facial cues.

The Advantage of Using an Optical See-Through Head-Mounted Display in Ultrasonography-Guided Needle Biopsy Procedures: A Prospective Randomized Study.

An optical see-through head-mounted display (OST-HMD) can potentially improve the safety and accuracy of ultrasonography (US)-guided fine-needle aspiration. We aimed to evaluate the usefulness of an OST-HMD in US-guided needle-puncture procedures. We conducted a prospective randomized controlled study in which we compared the accuracy and safety of the US-guided needle puncture procedure and the stress on the practitioner when using OST-HMD versus standard US display (SUD). Inexperienced medical students were enrolled and randomly divided into two groups. A breast phantom was used to evaluate the required time and accuracy of the US-guided needle puncture. Practitioner stress was quantified using a visual analog scale (VAS). When the procedure was performed for the first time, the time required to reach the target lesion at a shallow depth was significantly shorter in the OST-HMD group (39.8 ± 39.9 s) than in the SUD group (71.0 ± 81.0 s) (p = 0.01). Using the OST-HMD significantly reduced the unintentional puncture of a non-target lesion (p = 0.01). Furthermore, the stress felt by the practitioners when capturing the image of the target lesion (p < 0.001), inserting and advancing the needle more deeply (p < 0.001), and puncturing the target lesion (p < 0.001) was significantly reduced in the OST-HMD group compared with that in the SUD group. Use of OST-HMD may improve the accuracy and safety of US-guided needle puncture procedures and may reduce practitioner stress during the procedure.

Quantitative evaluation of stereoscopic augmented reality visualization using an optical see-through head-mounted display

Purpose The objectives of the current study were to propose a calibration method for accurate augmented reality visualisation using an optical see-through head-mounted display, and to qualitatively evaluate visualisation accuracy for the application in computer assisted surgery. Methods An adaptation of stereo single-point active alignment method was proposed as the calibration procedure together with verification Three tests were performed: display of points in 3D space, on the plane, and on the surface of a skull phantom on inexperience users ( 5 participants) and experienced users ( 17 participants). Results The highest accuracy of visualization was obtained for skull phantom visualisation for an inexperienced user (3.00 mm, std 0.75 mm), while the lowest accuracy was obtained in a 3D visualisation test for an inexperienced user (22.95 mm, std 18.04 mm). The largest error was related to the depth component and amounted to 18.49 mm, std 18.10 mm for an inexperienced user. Conclusions It is possible to achieve relatively high visualisation accuracy (less than 5 mm for visualisation in space) for selected users but providing it for the group of inexperienced users seems to remain a major challenge. The accuracy of point indication can be substantially and statistically significantly increased by visualising objects on surfaces. The proposed methods and obtained results can serve as a basis for further implementation of augmented reality visualisation on an optical see-through head-mounted display in applications requiring high-quality augmented reality guidance of manual tasks.

Effects of Reading Text on an Optical See-Through Head-Mounted Display During Treadmill Walking in a Virtual Environment

This study evaluated the effects of reading text on an optical see-through head-mounted display during treadmill walking in a virtual environment. Thirty participants were recruited to perform Chinese reading tasks by reading six types of text layouts in two polarity settings. A virtual environment was displayed on a projection screen to simulate a walking situation in which obstacles were encountered. High- and low-density obstacles were applied. Objective measures included reading performance, gait variability, and the error count of obstacles. The subjective measures were the situational awareness rating technique (SART) and NASA-Task Load Index (NASA-TLX) scores. The results indicated that font size and polarity affected the gait variability, reading performance, situational awareness, and mental workload ratings. The obstacle density affects gait variability and situation awareness rating. The layout with 32-point text and positive polarity yielded faster reading speed, better subjective ratings, and less variability in minimum foot clearance.

Freehand Gestural Selection with Haptic Feedback in Wearable Optical See-Through Augmented Reality

Augmented reality (AR) technologies can blend digital and physical space and serve a variety of applications intuitively and effectively. Specifically, wearable AR enabled by optical see-through (OST) AR head-mounted displays (HMDs) might provide users with a direct view of the physical environment containing digital objects. Besides, users could directly interact with three-dimensional (3D) digital artefacts using freehand gestures captured by OST HMD sensors. However, as an emerging user interaction paradigm, freehand interaction with OST AR still requires further investigation to improve user performance and satisfaction. Thus, we conducted two studies to investigate various freehand selection design aspects in OST AR, including target placement, size, distance, position, and haptic feedback on the hand and body. The user evaluation results indicated that 40 cm might be an appropriate target distance for freehand gestural selection. A large target size might lower the selection time and error rate, and a small target size could minimise selection effort. The targets positioned in the centre are the easiest to select, while those in the corners require extra time and effort. Furthermore, we discovered that haptic feedback on the body could lead to high user preference and satisfaction. Based on the research findings, we conclude with design recommendations for effective and comfortable freehand gestural interaction in OST AR.

Color Contrast Enhanced Rendering for Optical See-Through Head-Mounted Displays.

Most commercially available optical see-through head-mounted displays (OST-HMDs) utilize optical combiners to simultaneously visualize the physical background and virtual objects. The displayed images perceived by users are a blend of rendered pixels and background colors. Enabling high fidelity color perception in mixed reality (MR) scenarios using OST-HMDs is an important but challenging task. We propose a real-time rendering scheme to enhance the color contrast between virtual objects and the surrounding background for OST-HMDs. Inspired by the discovery of color perception in psychophysics, we first formulate the color contrast enhancement as a constrained optimization problem. We then design an end-to-end algorithm to search the optimal complementary shift in both chromaticity and luminance of the displayed color. This aims at enhancing the contrast between virtual objects and the real background as well as keeping the consistency with the original displayed color. We assess the performance of our approach using a simulated OST-HMD environment and an off-the-shelf OST-HMD. Experimental results from objective evaluations and subjective user studies demonstrate that the proposed approach makes rendered virtual objects more distinguishable from the surrounding background, thereby bringing a better visual experience.

Robotic Telepresence Based on Augmented Reality and Human Motion Mapping for Interventional Medicine

Robotic telemedicine has a high potential to promote the development of future medicine. It enables the treatment of critical patients in geographically remote locations in time, and also particularly useful for avoiding infection risks during pandemics of infectious diseases. However, visual feedback from the patient’s side to the clinician’s side is often unintuitive due to a lack of depth information, occlusion, and viewing angles. This paper develops a robotic telepresence method based on augmented reality and human motion mapping for interventional medicine. An optical see-through head-mounted display (OST-HMD) is mounted at the distal end of a redundant manipulator, which can provide views from different directions, and the clinician can see through key anatomies and instruments located inside the patient body. The control of the manipulator is accomplished via human motion mapping, with both the end-effector and manifold of the human arm being mapped to the manipulator. A simple evaluation method is developed to assess the accuracy of head localization of the OST-HMD. Telepresence experiments are carried out to verify the effectiveness of the proposed method, with which the operator can see through a human airway phantom.